<html>
<head><meta charset="utf-8"><title>Lexer · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html">Lexer</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="186345193"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186345193" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> std::Veetaha <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186345193">(Jan 23 2020 at 00:03)</a>:</h4>
<p>Are we going to stick with using <code>rustc_lexer</code> crate in future or is this temporary?</p>



<a name="186345422"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186345422" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186345422">(Jan 23 2020 at 00:06)</a>:</h4>
<p>This is the opposite of temporary</p>



<a name="186345432"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186345432" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186345432">(Jan 23 2020 at 00:07)</a>:</h4>
<p>We probably should tweak our integration with <code>rustc_lexer</code></p>



<a name="186345481"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186345481" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186345481">(Jan 23 2020 at 00:08)</a>:</h4>
<p>and we <em>might</em> tweak the interface and scope of <code>rustc_lexer</code> (things like lowering an integer literal to an integer value ideally should be in <code>rustc_lexer</code>), but the overall structure should be roughly as it is today</p>



<a name="186345651"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186345651" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> std::Veetaha <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186345651">(Jan 23 2020 at 00:10)</a>:</h4>
<p>And what would you want to tweak in our integration with <code>rustc_lexer</code> today?</p>



<a name="186345860"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186345860" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186345860">(Jan 23 2020 at 00:14)</a>:</h4>
<p>At least not dropping errors on the floor. Quite probably something else: the current contents of <code>lexer.rs</code> is basically the minimal diff which was possible to apply to thrown away our previous bespoke lexer and replace it with <code>rustc_lexer</code>. For example, </p>
<div class="codehilite"><pre><span></span><span class="w">            </span><span class="n">rustc_lexer</span>::<span class="n">TokenKind</span>::<span class="n">Ident</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">                </span><span class="kd">let</span><span class="w"> </span><span class="n">token_text</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="n">text</span><span class="p">[..</span><span class="n">rustc_token</span><span class="p">.</span><span class="n">len</span><span class="p">];</span><span class="w"></span>
<span class="w">                </span><span class="k">if</span><span class="w"> </span><span class="n">token_text</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">&quot;_&quot;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">                    </span><span class="n">UNDERSCORE</span><span class="w"></span>
<span class="w">                </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">                    </span><span class="n">SyntaxKind</span>::<span class="n">from_keyword</span><span class="p">(</span><span class="o">&amp;</span><span class="n">text</span><span class="p">[..</span><span class="n">rustc_token</span><span class="p">.</span><span class="n">len</span><span class="p">]).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="n">IDENT</span><span class="p">)</span><span class="w"></span>
<span class="w">                </span><span class="p">}</span><span class="w"></span>
<span class="w">            </span><span class="p">}</span><span class="w"></span>
</pre></div>


<p>immediately jumps at me as something we might want to change at either our or <code>rustc_lexer</code> side</p>



<a name="186345880"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186345880" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186345880">(Jan 23 2020 at 00:15)</a>:</h4>
<p>I don't have time right now to investigate what <em>exactly</em> needs to be changed and how :D</p>



<a name="186346295"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186346295" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> std::Veetaha <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186346295">(Jan 23 2020 at 00:23)</a>:</h4>
<p>Yeah, I am currently working on that not-dropping-errors-on-the-floor issue. But didn't know we had another lexer implementation it the past. Regarding that UNDERSCORE token, I'll also look at it, but after I am done with errors preserving.</p>



<a name="186347014"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186347014" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> std::Veetaha <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186347014">(Jan 23 2020 at 00:36)</a>:</h4>
<p>And by <em>"lowering an integer literal to an integer value"</em> do you mean parsing the integer token string representation like <a href="https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/ra_syntax/src/ast/extensions.rs#L26" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/ra_syntax/src/ast/extensions.rs#L26">here</a>? I also wonder whether this is legal at all (I mean there may be some rust-specific features of int literals that <code>str::parse::&lt;usize&gt;()</code> may not understand...)</p>



<a name="186347477"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186347477" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186347477">(Jan 23 2020 at 00:44)</a>:</h4>
<p>That code is specifically for tupple suffixes, so it is likely more or less<br>
ok. I mean something that makes sure that 4_2f32 is 42.0</p>



<a name="186348020"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186348020" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> std::Veetaha <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186348020">(Jan 23 2020 at 00:55)</a>:</h4>
<p>Hmm, it's hard to find a place where we parse numeric literals value... And I don't see where it is implemented now.</p>



<a name="186369409"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186369409" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186369409">(Jan 23 2020 at 09:03)</a>:</h4>
<p>we don't do that yet, but rustc does</p>



<a name="186373390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186373390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> std::Veetaha <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186373390">(Jan 23 2020 at 10:01)</a>:</h4>
<p>Okay, I guess we would need to expose that as a public API on <code>rustc_lexer</code> crate. Today I'll create an issue for that at their repo in order not to forget about it.</p>



<a name="186373515"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186373515" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186373515">(Jan 23 2020 at 10:02)</a>:</h4>
<p>I think it would be much easier forget about the fact that the issue exists than about the fact that we need to refactor this eventually</p>



<a name="186373819"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186373819" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> std::Veetaha <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186373819">(Jan 23 2020 at 10:06)</a>:</h4>
<p>I add issues into my bookmarks, so won't forget about it)</p>



<a name="186432659"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Lexer/near/186432659" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> std::Veetaha <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Lexer.html#186432659">(Jan 23 2020 at 20:37)</a>:</h4>
<p>Okay, so I created <a href="https://github.com/rust-lang/rust/issues/68492" target="_blank" title="https://github.com/rust-lang/rust/issues/68492">an issue at rustc repo</a>, I hope someone will answer.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>